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Abstract 

This study explored how teachers interpreted and responded to their own student work during the 
process of formative assessment. The study involved a purposefully selected sample of 32 teachers in 
grades K-5 who had been trained by the Ongoing Assessment Project (OGAP) to use learning 
progressions to analyze and respond to evidence in student work. Since formative assessment is 
fundamentally an interpretive process, involving continually eliciting and interpreting evidence of student 
thinking from student work in order to inform teaching and learning (Black & Wiliam, 2009), the study 
analyzed data collected through semi-structured interviews. 


The study found variations in the way teachers make sense of their student work for formative 
assessment that were related to their underlying goals for student learning. Teachers with an 
achievement orientation tended to focus on performance goals: giving formative assessment items to 
gauge student performance on problems that reflected what had recently been taught and focused on 
singular or multiple components of performance to make a binary judgment (i.e. students who “get it or 
don't get it”). Teachers with a learning orientation gave items to learn more about what students were 
able to do on different types of problems and focused on student strategies as an indicator of underlying 
understanding and development. These orientations also had implications for the instructional response 
teachers developed; as teachers looked beyond surface features of student work and binary distinctions, 
they developed more differentiated responses that built on students’ knowledge and their ability to 
develop more sophisticated understanding. In between these two extremes, we found three categories of 
hybrid approaches to formative assessment, demonstrating a push-and-pull between achievement and 
learning orientations at different decision points during the steps of the formative assessment process. 
Those decision points — the teachers’ purpose in giving an item, the evidence focused on, the interpretive 
framework used to analyze the evidence, and the focus of the instructional responses — offer multiple 
footholds in the formative assessment process where teachers can begin to try out new approaches that 
reflect a shift in orientation to student learning. 


The study shows that using formative assessment is not simply a matter of taking up new practices and 
using new tools. The variations in understanding and use of the ideas that were offered in professional 
development, as reflected in teachers’ actual practices, suggests that it is important to provide 
opportunities for sustained learning and supported use over time. 
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Abstract 


This study explored how teachers interpreted and responded to their own student work 
during the process of formative assessment. The study involved a purposefully selected 
sample of 32 teachers in grades K-5 who had been trained by the Ongoing Assessment 
Project (OGAP) to use learning progressions to analyze and respond to evidence in 
student work. Since formative assessment is fundamentally an interpretive process, 
involving continually eliciting and interpreting evidence of student thinking from student 
work in order to inform teaching and learning (Black & Wiliam, 2009), the study analyzed 
data collected through semi-structured interviews. 

The study found variations in the way teachers make sense of their student work for 
formative assessment that were related to their underlying goals for student learning. 
Teachers with an achievement orientation tended to focus on performance goals: giving 
formative assessment items to gauge student performance on problems that reflected 
what had recently been taught and focusing on singular or multiple components of 
performance to make a binary judgment (i.e. students who “get it or don’t get it”). 
Teachers with a learning orientation gave items to learn more about what students 
were able to do on different types of problems and focused on student strategies as 
an indicator of underlying understanding and development. These orientations also 
had implications for the instructional response teachers developed; as teachers looked 
beyond surface features of student work and binary distinctions, they developed more 
differentiated responses that built on students’ knowledge and their ability to develop 
more sophisticated understanding. In between these two extremes, we found three 
categories of hybrid approaches to formative assessment, demonstrating a push-and-pull 
between achievement and learning orientations at different decision points during the 
steps of the formative assessment process. Those decision points — the teachers’ purpose 
in giving an item, the evidence focused on, the interpretive framework used to analyze 
the evidence, and the focus of the instructional responses — offer multiple footholds in the 
formative assessment process where teachers can begin to try out new approaches that 
reflect a shift in orientation to student learning. 

The study shows that using formative assessment is not simply a matter of taking 
up new practices and using new tools. The variations in understanding and use of the 
ideas that were offered in professional development, as reflected in teachers’ actual 
practices, suggests that it is important to provide opportunities for sustained learning and 
supported use over time. 
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Introduction 


Recent research on mathematics teaching promotes a view of ambitious instruction 
that calls for teachers to regularly elicit, make sense of, and respond to students’ 
mathematical thinking and problem-solving strategies (Lampert, Beasley, Ghousseni, & 
Kazemi, 2010; Sztajn, Confrey, Wilson & Edgington, 2012). Although there is a growing 
body of work in mathematics education around teacher noticing of student thinking 
during instruction (e.g., Schack, Fisher & Wilhelm, 2017; Sherin, Jacobs & Phillips, 2010), 
less attention has been paid to how teachers interpret and respond to their own student 
work to inform instruction. 

Formative assessment is an instructional practice that involves continually eliciting 
and interpreting data, or evidence of student thinking, from student work in order to 
inform teaching and learning (Black & Wiliam, 2009). Existing studies of teachers’ use of 
student work to inform instruction highlight that teachers tend to focus on what students 
are or are not able to do, rather than on what they understand (Ebby & Sirinides, 2015; 
Christman, et al., 2009; Goertz, Olah, & Riggan, 2009; Kazemi & Franke, 2004; Supovitz, 
Ebby & Sirninides 2013), or explain student performance in terms of pre-existing beliefs 
or judgments about their ability (Horn, 2007; Jackson, Gibbons, & Sharpe, 2017; Wilson, 
Sztajn, Edgington, Webb, Myers, 2017). 

This paper adds to this growing knowledge of teachers’ interpretations of student 
thinking by exploring how teachers interpret and respond to student work during the 
process of formative assessment and how their interpretations reflect underlying views 
of learning. We examine teachers’ understandings of formative assessment through the 
lens of how teachers sorted, interpreted, and responded to their own students’ work after 
being introduced to a learning trajectory for formative assessment through the Ongoing 
Assessment Project (OGAP). Learning trajectories are “empirically supported hypotheses 
about levels or waypoints of thinking, knowledge, and skill in using knowledge, that 
students are likely to go through as they learn mathematics” (Daro, Mosher, & Corcoran, 
2011, p. 12). They can enhance the formative assessment process by providing a road 
map for setting goals, as well as analyzing and responding to evidence of student 
thinking (Ebby & Petit, 2017; Heritage, 2008). 


The questions guiding our analysis include: 


1. What are the variations in the way teachers make sense of their student 
work for formative assessment? 


2. What do teachers’ interpretations of and responses to student work 
reveal about their underlying goals for student learning? 


3. What are the implications for teachers’ use of learning trajectory- 
oriented formative assessment? 


In the following sections, we describe empirical and conceptual foundations for the 
OGAP formative assessment process, which teachers were introduced to during the 
summer prior to this study. We then describe the theoretical framework that informed our 
analysis of teachers making sense of, interpreting, and responding to student work during 
the formative assessment process, and more specifically the role played in that process 
by teachers’ underlying goals for student learning. 
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Learning Trajectory Informed Formative Assessment 


In recent years formative assessment has been promoted as one of the most impactful 
educational interventions in terms of improving student achievement, particularly in 
elementary mathematics (Black & Wiliam, 1998; Klute, Apthorp, Harlacher & Reale, 2017). 
In contrast to grading or judging student performance after learning has taken place, 
formative assessment involves providing feedback and adjusting instruction continuously. 
Effective formative assessment is a complex process, involving the following steps: 


1. Identifying the learning goal. 


2. Eliciting information about what students currently know and understand 
in relation to the learning goal. 


3. Analyzing the student response for evidence of students’ developing 
knowledge and understanding. 


4. Identifying the gap between the learning goal and where the learner is 
currently. 


5. Providing feedback to students and/or developing an instructional 
response that moves the learner closer to the learning goal. 


In distinguishing formative from summative assessments, many people erroneously focus 
on the frequency, timing, or format of the assessments themselves. In contrast, Black and 
Wiliam (2009) propose that an assessment is formative based on its function: 


Practice in a classroom is formative to the extent that evidence about student 
achievement is elicited, interpreted, and used by teachers, learners, or their 
peers, to make decisions about the next steps in instruction that are likely to 
be better, or better founded, than the decisions they would have taken in the 
absence of the evidence that was elicited. (Black & Wiliam, 2009) 


According to this definition, it is the interpretation and response to the evidence that 
is elicited by the assessment that is central to making it formative. Because formative 
assessment is fundamentally an interpretive process for both teachers and students, the 
conceptual frameworks teachers use to make sense of and respond to the evidence are 
of critical importance. While teachers cannot directly observe student understanding, 
they can draw from behaviors and evidence of students’ mathematical thinking to build 
a model of student knowledge (Mojica & Confrey, 2009; von Glasersfeld, 1995). The 
teachers in this study had been introduced to research-based conceptual frameworks 
that spelled out a learning progression that children move through when developing 
understanding of core mathematics topics. 

Heritage (2011) proposes that learning progressions can enhance the formative 
assessment process by providing a guide for teachers to formulate clear learning goals 
as well as to analyze and respond to evidence in student work: 


To do this effectively, teachers need to have in mind a continuum of how 
learning develops in any particular knowledge domain so that they are able to 
locate students’ current learning status and decide on pedagogical action to 
move students’ learning forward. Learning progressions that clearly articulate a 
progression of learning in a domain can provide the big picture of what is to be 
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Figure 1. OGAP Multiplication Progression (Petit, Hulbert & Laird, 2017) 
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learned, support instructional planning, and act as a touchstone for formative 
assessment (p. 1) 


Similarly, Szjatin et al. (2012) propose that learning trajectories, the more commonly used 
term for learning progressions in mathematics education, can help teachers be “guided 
by the logic of the learner” in eliciting evidence of student learning and providing 
feedback to learners by supporting teachers in “examining the boundaries of what 
students do and do not understand” (p. 152). 


The Ongoing Assessment Project 


The notion that formative assessment can be enhanced by learning progressions is 
central to the Ongoing Assessment Project (OGAP) (Ebby & Petit, 2018). The OGAP 
Progressions describe learning trajectories in a visual format of typical strategies that 
students use to solve problems in several core content areas (additive, multiplicative, 
fractional, and proportional thinking) (see Appendices B and C). These strategies 

are organized into developmental levels, describing the movement from less to more 
sophisticated strategies as student understanding deepens. For example, Figure 1 shows 
how students progress in multiplication from additive to multiplicative reasoning, initially 
solving problems by counting equal groups by ones, then moving to repeated addition, 
skip Counting, and using area models to eventually develop more abstract strategies 
based on their understanding of place value and properties of multiplication. The array 
and area models in the transitional level represent an important developmental bridge 
from additive to multiplicative thinking. Evidence suggests that the development of 
student strategies does not occur in a strictly linear process; rather students’ use of 
strategies tends to move back and forth across the levels as they encounter different 
problem structures and contexts until ultimately stabilizing into efficient and flexible 
strategies (Hulbert, Petit, Ebdby, Cunningham & Laird, 2017). The progressions are designed 
to be useful tools for teachers, helping them to identify levels of student thinking based 
on the strategies they use to solve particular problems, as well as offering instructional 
guidance for transitioning student understanding and strategies from one level to the 
next. 

A core tenet of OGAP is that student strategies offer a window into their developing 
understanding. A second core tenet of the program is that teachers can learn more 
about the depth of student understanding by giving formative assessment items regularly 
and ensuring that those items represent the range of problem types and structures in the 
content area. For this reason, OGAP provides a bank of formative assessment items for 
each content area, which are intentionally engineered to elicit student understandings 
and common misconceptions on a range of problem types and structures. In OGAP 
professional development, teachers are introduced to the research on students’ 
understanding of mathematical content they are teaching. They also practice using 
the progressions to analyze samples of student work on OGAP items by sorting them into 
different piles by strategy and common understandings. 

Once teachers are introduced to the research and instructional tools in professional 
development, they are expected to give and analyze items regularly to inform their 
instruction. In addition, schools are encouraged and supported in having teachers meet 
regularly in grade-level groups to collaboratively analyze and respond to student work. 
In this study we focused on how teachers made sense of the formative assessment 
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process, from the administration of an assessment item to the instructional response 
based on the evidence. What question about student learning were teachers trying 

to answer when they gave a formative assessment item? How did this purpose guide 
the choice of the item they gave? What evidence did teachers pay attention to in the 
student work and how did they make sense of and interpret this evidence in relation 

to student learning and instruction? We were also interested in understanding whether 
and how these interpretations were related to teachers’ underlying views about student 
learning. In our early stages of analysis, it became evident that the way that teachers 
conceptualized learning goals for their students played an integral role in shaping the 
interpretation process. 


Theoretical Foundation: Expectations 
and Goals for Student Learning 


In considering how teachers conceptualized goals for student learning, we draw on 
two bodies of research: cognitive research within mathematics education that offers 
detailed descriptions of children’s learning trajectories in core mathematics domains 
(discussed above), and research on expectations, beliefs, and mindsets, which shape 
teachers’ perceptions of students’ capabilities and the learning goals they establish 
(e.g., Dweck, 2008; Ferguson, 1998; Jackson et al., 2017). Since all teachers in this 

study had been introduced to a learning-trajectory approach to student learning and 
formative assessment, we were interested in the extent to which teachers took up and 
understood or modified this approach in practice when interpreting student work. With 
respect to teacher expectations, there is strong evidence that teachers’ beliefs about 
their students’ abilities influence the learning opportunities they make available to them 
in the classroom (e.g., Diamond, Randolph, & Spillane, 2004; Jackson et al., 2017; Sztajn, 
2003). Sztajn, for example, found that teachers’ expectations were manifest in the types 
of learning goals they held for their students, based on their beliefs about students’ 
needs, and that these were in turn associated with students’ socioeconomic background 
and the resulting opportunities to learn. 


Teachers who view intelligence and ability as fixed are likely to hold learning goals 
for their students that focus on performance, whereas teachers who view intelligence 
as malleable are more likely to view learning as a process of development. 


In our analysis of teachers’ interpretations of students’ work, findings from research on 
goal orientation also emerged as an important and explanatory factor for how teachers 
made sense of and used the learning progression. We found Dweck and Legget’s (1988) 
distinction between performance goals, which focus on documenting fixed ability, 
and learning goals, which focus on developing or increasing ability over time, to be 
helpful in characterizing differences that emerged in our data in the goals teachers 
had for assessing student learning. Dweck’s (1986) work on growth and fixed mindsets is 
best known for illuminating the impact of an individual's beliefs and goals on their own 
learning. Learners who have a fixed mindset believe that intelligence and ability are 
fixed, innate traits, which individuals can do little to change. From this view, performance 
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is an indication of one’s capabilities. In contrast, individuals with a growth mindset 
believe that intelligence and ability are malleable and can be developed through hard 
work and education (Dweck, 2008). Learners with a growth mindset show persistence 
and seek challenge, even in the face of perceived low ability or performance. 

We see a strong resonance between Dweck's (1986; 2008) conception of growth 
mindset and the developmental perspective underlying learning trajectories. A growth 
mindset is premised on the idea that learning is incremental and occurs over time, 
propelled forward by sustained effort, strategy development, and mentoring. Learning 
trajectories, which map development of understanding in particular domains, from 
beginning or basic understanding to increasingly more sophisticated and understanding, 
chart the paths this growth is likely to take (Heritage, 2011; Sztajn, 2012). Simply put, a true 
learning trajectory approach assumes a growth mindset. 

Although Dweck’s (1986) framework focused on learners’ goal orientations, this 
work has implications for how teachers conceptualize goals for their students’ learning. 
Teachers who view intelligence and ability as fixed are likely to hold learning goals for 
their students that focus on performance, whereas teachers who view intelligence as 
malleable are more likely to view learning as a process of development and are likely 
to set goals for students that map onto some sort of learning progression. The types of 
goals teachers set for students can impact how they interpret students’ work and guide 
their future growth. As Elliot and Dweck (1988) put it, goals generate their “own set of 
concerns” and create their “own framework for processing incoming information” (p. 
5). Given the fact that identifying the learning goal is the first step of the formative 
assessment process, it makes sense that the goals teachers hold for their students act 
as a framework for making sense of the evidence they collect of student learning as 
well as the way that they respond to that evidence. Drawing on this work, we consider 
how different types of goals teachers hold for their students’ learning work in practice to 
create a unique framework for processing evidence of student thinking in the formative 
assessment process. 


Methods 


Research Context 


This study was conducted in 2017-18 in the context of the second year of a three- 

year rollout of OGAP implementation in the School District of Philadelphia, a large 
economically and ethnically diverse public system. During this phase, 10 elementary 
schools signed up to send 116 teachers to a five-day OGAP training for the first time. 

An additional 19 elementary schools, whose teachers had participated in previous 
trainings, sent 101 new teachers to the training. Although there were additional perks 
for new schools who sent at least 65% of their teaching staff to training, attendance 
varied greatly by school. The training was comprised of five full-day workshops organized 
by grade band (K-2, 3-5, 6-8) and focused on conceptual understanding of the core 
content area, research on student learning of the core concepts, problem structures, 
visual models, properties of operations, understanding the Common Core standards, and 
using learning progressions to sort and analyze student work on formative assessment 
items and pre-assessments. During the school year, schools also received ongoing 
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support for implementing OGAP in the form of school visits by OGAP trainers and 
invitation to voluntary, follow-up workshops. Schools varied in the extent to which they 
took up these opportunities and in the leadership styles of the principals and other school 
leaders in supporting implementation of OGAP locally (Flack, Morrison, Hess, Kolouch & 
Pierce, forthcoming). 


Data Collection 


This study is part of a larger qualitative inquiry that explored how teachers took up and 
made sense of learning trajectory-oriented formative assessment in a range of school 
contexts. Data were collected through a series of three semi-structured interviews 
conducted with a purposeful sample of 32 grade K-5 teachers from nine schools who had 
completed their first year of OGAP training. This study drew primarily on data collected 
from the second of those interviews. 


Participants 


To identify participants for the study, all teachers who attended the OGAP summer 
training for the first time in the summer of 2017 were surveyed on their willingness to 
participate. From the subset of teachers that responded, a set of elementary schools was 
identified that had at least two OGAP-trained teachers at each grade band (K-2 and 
3-5). Based on their responses, 32 teachers across eight schools agreed to participate in 
the study, with between one and six teachers at each school. Five of the eight schools 
were in the first year of implementing OGAP. Although all teachers in the sample had just 
completed OGAP training for the first time, there were three schools that were already 
using OGAP in grades 3-5. As a result, five teachers in our sample may have had some 
prior exposure to analyzing student work on OGAP items in grade-level meetings, even 
though it was their first time attending OGAP training. The schools represented a range of 
school sizes and demographics (shown in Table 1) and a range of performance profiles 
as determined by the district to represent student achievement, progress, and climate; 
out of four possible tiers, one school was in the lowest, tier 1, three in tier 2, and five in 
tier 3. Our selection criteria yielded a varied sample of participants from grades K-5 with 
teaching experience ranging between two and 36 years, with a mean of 15 years. 

Teachers in the study were interviewed three times throughout the course of the 
school year, in the fall, winter, and spring. Each interview was designed to understand a 
different component of how teachers were taking up and making sense of OGAP and 
the learning-trajectory approach to formative assessment. The first interview focused on 
baseline use of formative assessment, views of students, and view of math instruction 
generally. The second interview focused on teachers’ analysis and sense-making of a set 
of their own students’ work on a formative assessment item. Teachers selected the item 
to bring to the interview and were asked to sort and analyze it as they normally would. 
The final interview partially mirrored the first, focusing on general formative assessment 
use, teachers’ views of students, and their overall understanding of the trajectory-based 
formative assessment training they received. 

Data for this study was taken primarily from the second interview as it provided 
evidence of the sense-making teachers were doing with their own students’ work. Before 
the second interview, teachers were asked to administer a formative assessment item. 
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In the interview, they were then asked to (1) explain how they sorted their student work, 
(2) analyze representative examples from each pile they created, and (3) formulate 
instructional responses for each example as well as for the class as a whole. 


Data Analysis 


Analysis began by producing an analytical memo to document each teacher's 
categorization of student work and then looking across these memos to develop initial 
categories of sorting strategies. We then used a constant comparative approach to 
apply and continually refine these categories into a typology (Strauss & Corbin, 1990). 
After a typology was developed, the research team continued to refine the categories 
by rereading teacher interviews from each category to both better understand 

the commonalities of teachers in each category and the major delineations across 
different categories. All interviews were double-coded, and any disagreements were 
reconciled through discussion. This iterative engagement led to the identification of 
overarching foci and interpretations teachers in each category made throughout the 
formative assessment process. As the analysis proceeded, teachers’ underlying views of 
learning and the guiding purposes for analyzing student work emerged as being salient 
dimensions that further informed our conceptual framework. 


Table 1: School Demographics for Participant Sample 


School 


Ti moO w 


Total 


School Econ. Black/AA —Hispanic/ White Asian = Multi-race ELL Number of Participants 
Size ** Disad. Latino 
Trained in Trained in 
Additive Multiplicative 
K ] D 3 4 5 
sm 100% 90% 1% 2% 0% 6% 0% 2 1 
Md 100% 45% 19% A% 26% 9% 21% 1 1 
Lg 72% 17% 13% 52% 8% 10% 7% 1 ] 2 
sm 100% 82% 5% 3% 3% 6% % ] 1 
sm 76% 21% 16% 44% 6% 12% 8% 2 2 
sm 100% 10% 14% 62% 1% 12% % 
Md 100% 7% 49% 6% 33% 6% 41% Te 2 
Md 22% 9% 6% 64% 8% 13% To 1 1 i* 
Md 100% 24% 62% 5% 1% 7% 13% 1 2. 2 2 
2 8 7 9 4 2 


*Taught a mixed class with the grade above, i.e., Ist and 2nd or 3rd and 4th 
**School size determined by enrollment as of Oct. 2017. Small: less than 500; Medium: 500-1000, Large more than 1000 


12 
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Findings: Pathways to Making Sense of Student Work 


Our analysis yielded a typology of five different approaches to using student work to 
inform instruction that are influenced by different goals for both formative assessment 
and student learning. All 32 teachers in the study participated in the OGAP professional 
development in the summer prior to the study, which focused on developing 
understanding of the core content and using a learning trajectory to sort, interpret, and 
respond to student work in relation to a progression of understanding. Still, our analysis of 
their approaches to this process surfaced substantial variation in what teachers attended 
to and how they interpreted their own work. 
We initially distinguished between two distinct and contrasting approaches: one that 
followed what we call an achievement orientation and one that followed a learning 
orientation. The learning orientation, which reflects the approach underlying the OGAP 
training, was characterized by teachers who used students’ work on the problem as 
a vehicle to surface features of student understanding, focused on student strategies 
as a reflection of developing understanding, and situated that evidence within a 
developmental progression to determine an instructional response. This orientation 
aligns with what Dweck and Legett (1988) refer to as learning goals, as it is focused on 
developing or increasing ability over time. In contrast, the achievement orientation was 
represented by teachers who focused on performance goals, by considering student 
performance on the item as an end in itself, attending to accuracy and correctness 
to make a binary judgment (right or wrong, gets it or doesn't get it). The achievement 
orientation resulted in an instructional focus on correctness of student performance. 
As characterized by Elliot and Dweck (1988), these orientations established their 


Table 2: Contrasting Characteristics of the Achievement and Learning Orientations as Applied to Student 
Learning and Formative Assessment 


CONTEXT 


Student 
Learning 


Formative 
Assessment 


DIMENSIONS 


Mindset 
Goals 


Focus 


Purpose 


Analysis 


Interpretation 


Instructional Response 


ACHIEVEMENT ORIENTATION 


Fixed mindset 
Performance Goals 


Mastery of performance 


Gauge mastery of skill, concept, 


or problem 


Focus on accuracy and 
correctness 


Using evidence to make binary 
judgment 


Focus on correction or 
remediation to achieve desired 
performance 


LEARNING ORIENTATION 


Growth mindset 
Learning Goals 


Improvement of understanding 


Elicit evidence of understanding 


Focus on strategies 


Situating evidence within 
developmental progression 


Focus on building on current 
understanding to get to next level of 
strategy or understanding 
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Figure 2. Five assessment pathways taken by teachers as they went own “sets of concerns” 
through the steps of formative assessment (p. 5) and resulted in 
different frameworks 


through which teachers 
Goal Orientation made sense of students’ 
work. In the context of 
formative assessment, 
Achiev these frameworks included 
Mastery of p R=YeT dal late} differences in the purpose 
Ten)e)cohd-Vant-al med mUlate (Ved relate Late] of the formative assessment 
and how teachers analyze, 
A. Guiding question... interpret, and respond 


to their student work 
Can they do it? (summarized in Table 2). 
We also identified three 
™ ey additional approaches 
that did not fit squarely 


B. Sorting student work by... into an achievement or 
a learning orientation, 
Answer resent Strategies and instead reflected 


hybridized elements of both 


i 4 aE ev y orientations at different 


steps of the formative 

C. Interpreting evidence in rslaton to... assessment process. Figure 2 
illustrates the five pathways 
we identified to describe 

the frameworks used by 
sb % Y 4 sy aoe teachers to make sense of 
student work in the formative 
assessment process. 


1BY-W7=) [e) o}aal=yaicel| 


Binary judgment progression 


D. Formulating an instructional response focused on... 


Correction Different Building on 


The pathways are comprised of four key decision points: a) the guiding 
question or purpose of giving the formative assessment item; b) sorting the 
student work based on evidence; c) interpreting that evidence in relation to an 
underlying view of learning; and d) formulating an instructional response. 


As the different pathways in Figure 3 illustrate, the purpose of giving a formative 
assessment item ranged from finding out whether or not students could solve a particular 
type of problem (“can they do it?”) fo a more open-ended investigation of finding 
out what they were able to do and wht the evidence suggested about what they 
understood. When looking at student work, teachers differed in whether they primarily 
attended to students’ answers, or the presence or accuracy of various components 
of the work, including the answer, solution strategy, and use of visual representations. 
Teachers also differed in the assessment lenses they used to interpret the work, from 
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making a binary judgment related to correctness to considering strategies in a 
developmental progression where one stage of understanding builds on the next. 

Finally, teachers varied in how they used their interpretations of students’ work to inform 
their instruction. Those following an achievement orientation focused on correcting 

or improving student performance. Those following a learning orientation tended to 
consider ways to improve understanding and support further development. Teachers who 
understood the developmental nature of the learning trajectory considered how to build 
on students’ current knowledge and understanding to move towards more sophisticated 
strategies and understanding. In the middle, some teachers considered levels of 

student strategies in an ordered continuum but didn’t show understanding of how these 
levels built on one another. (In Figure 3, part C, we refer this as a non-developmental 
progression.) Arrows illustrate these different paths taken by teachers at each step of the 
formative assessment process. Together these pathways comprised the analytical frame 
we used to make sense of our data. 

The names we use to identify each pathway reflect the major focus or emphasis 
guiding teachers’ analysis of student work in that group. Table 3 shows our categorization 
and names for each pathway and the frequency of each within our sample of 
32 teachers. In the following sections, we describe each of the five pathways as 
approaches to formative assessment, using data from teachers in each group to 
illustrate. More detailed and descriptive case studies of a single teacher from each 
category can be found in Appendix A. 


The Achievement Orientation to Formative Assessment 


Slightly more than half of the teachers in our sample demonstrated an approach to 
formative assessment that was aligned with an achievement orientation, in that they 
focused on student performance in relation to an external standard. However, there 
were some important differences in what they attended to and how they interpreted the 
work, resulting in three variations on their approach to formative assessment, described 
below. 


Correctness of Answer 


Seven of the teachers we interviewed focused on a correct answer on the formative 
assessment item as the primary indicator of student achievement in relation to a specific 
performance goal. Most often, their objective was to determine whether students had 
mastered a skill or strategy recently taught. Teachers in this group sorted their students’ 
work primarily by whether the numerical result was correct or incorrect, regardless of the 
strategy the child used to produce the answer or the nature of errors in the response. 
Some teachers also required an explanation in their consideration of correctness. In 
some cases, when the problem had more than one part, teachers sorted on whether the 
student obtained the correct answer on one or both parts of the problem, resulting in 
more than two piles, but in each case made binary distinctions within the piles. Sorting in 
this way ultimately resulted in making a binary distinction between students who “got it” 
and students did not. 

An illustrative example can be found in the analysis of student work on a subtraction 
problem by a second-grade teacher who sorted into two piles by correct or incorrect 
answer, despite the fact that students showed a wide range of strategies. When 
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questioned further, she noticed and identified different strategies used by students, 
including: “regrouping” (US traditional algorithm), number lines, base 10 visual 
models, “breaking apart” (decomposing by place value), and compensation. She 
did not, however, attend to the sophistication of thinking or underlying understanding 
represented in those strategies (e.g., whether they were counting by tens or ones on the 
number line, whether they used a model). She marked all papers containing the correct 
answer with a large C. As she said, “they either got it right or they didn’t get it right.” 

For some, like the following third grade teacher, sorting by correctness was a 
conscious decision to not use the OGAP progression: 


| was spending more time figuring out what level they were at and what that 
meant and it was frustrating to me... So | just do it where, “Are they right? Are they 
wrong? What did they do right? What did they do wrong?” And that informs my 
instruction from there. 


For others, sorting by correctness reflected their interpretation of OGAP. One teacher, 
for example, recalled that sorting by a “common theme” of understanding had been 
presented as an option at the OGAP training. She explained, “My theme was correct 
answer versus not correct answer.” 

In general, teachers who sorted primarily by correctness said that it helped inform 
their small group instruction, identify common mistakes, and determine if an intervention 
was needed. One teacher explained, “For small groups, | can see which group, if they 
totally didn’t get it at all, | know that I’m going to have to reteach that skill in a small 
group.” Teachers also described how it helped them get an overall sense of the class 
performance. Said one, “It shows me where the bulk of my kids are and whether they 
knew something or not.” Another said, “It shows me that even though we are... past this, 
they still aren’t completely grasping it.” 

In their proposed instructional responses, teachers focused on those students who 
were not demonstrating correct performance. When asked what she was planning to 
do as a result of giving a formative assessment, a second-grade teacher focused on the 
work in the incorrect pile: 


Maybe put them into small groups with all the same similar issues, like all these 
ones that subtracted up and just show them like on a whiteboard or with base 10 
blocks, the correct way of getting it and the correct answer, and, “Now look at 
your answer, what do you see here, can you tell why yours is wrong?” You know, 
just to get them to stop doing that. 


Underneath this sorting strategy, we found a view of learning as a process of improving 
performance by helping students learn how to solve problems correctly. Getting the 
correct answer provided an indication that students were ready to move on to the next 
topic, while unsuccessful performance required an intervention, most often in the form 
of small group instruction. The teachers in this group did not make use of the OGAP 
progressions and tended to use OGAP items in ways that seemed at odds with their 
intent. They treated the formative assessment items as a way to measure performance 
in relation to a specific learning goal (€.g., being able to solve a word problem 
involving two-digit subtraction) and saw a correct answer as the primary indicator that 
performance had been achieved. The thinking process that lay behind the use of the 
strategy or production of the answer was less important to them. 
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Table 3. Distribution of praticipating teachers by pathway 


ORIENTATION PATHWAY NUMBER OF TEACHERS 
Correctness of Answer 7 
Achievement Components of Performance 4 
Progression of Strategies 6 
Progression of Strategies 6 
Learning 
Progression of Understanding 9 
Total 32 


Components of Performance 


Four teachers who demonstrated an achievement orientation went beyond focusing 
ona single correct answer to consider multiple components or features of the response, 
including whether students could model the problem, show their thinking, or explain their 
answers. Like the previous group, teachers in this group wanted to determine whether 
students could successfully perform a previously taught skill or strategy to solve the 
problem correctly. At the same time, their aqoproach demonstrated an appreciation 

of the complexity of solving problems and the range of steps and strategies required. 

As a result, they did not look solely for the presence of a single correct answer, but 
considered multiple components, including some evidence of student understanding, 
resulting in multiple piles or categories of proficiency. For example, one second-grade 
teacher explained that she wanted students to practice “figuring out what the problem 
was talking about” and to “try to show a model.” Other teachers talked about whether 
students’ “reasoning was incorrect,” and tried to determine “their level of explaining” 

or created their own categories that indicated some focus on understanding. “I sort it 
into three piles, like right, wrong, and like close but confused,” said one. These examples 
illustrate that teachers in this group considered various dimensions of student work that 
went beyond simply producing the correct answer. 

Like the previous group, teachers in this group eventually made binary judgments 
(e.g., correct/incorrect, present/absent) for each component, resulting in piles that 
showed differences among the students but were not sequenced in any way and often 
attended to superficial aspects of student work instead of those revealing of student 
understanding. For example, one second-grade teacher focused on whether students 
used the correct operation, whether they used visual models or drawings to support their 
solution, and whether they had made errors in the use of the model. She felt that using 
models was important to understanding but did not focus on how students used the 
models or their underlying understanding of concepts. In addition, she often overlooked 
evidence of understanding (e€.g., a student who added up from the starting number 
was characterized as using the wrong operation) or misconceptions that the strategies 
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revealed (e.g., students who were using models to represent the answer rather than the 
problem situation). 

These teachers were also similar to the Correctness-of-Answer group in how they 
thought about the instructional implications of students’ work, in that they tended to 
focus on students who were not performing well in an effort to correct errors or show 
students the correct way to solve the problem. However, given the number of variations 
their sorting produced, their instructional responses tended to be more differentiated 
in relation to targeting student needs. They also formulated instructional responses that 
began to acknowledge the role of understanding, such as increased attention to visual 
models. 

In general, teachers who sorted by the presence of multiple components of 
performance said that it heloed them identify who had not reached the desired level of 
performance so that they could intervene or gauge the readiness of the class to move 
onin the curriculum. As one teacher in this group explained: 


If overall, everyone is doing what | need, then | know, okay, great, we can move 
on to something different. And if | only have a few kids that are struggling, that’s 
an easy fix. If most of them got it wrong, then | need to reteach that skill or there’s 
something that maybe they just didn’t understand. 


In this case, “what | need” included various aspects of student performance that went 
into producing a correct response. Successful performance in relation to these criteria 
was an indication that it was okay to move on to the next topic or skill, while unsuccessful 
performance required targeted intervention. 

This group was our smallest group (n=4) and could be considered an offshoot of the 
Correctness-of-Answers group. They brought similar goals to formative assessment and 
thought about learning in a similarly binary process of mastery of performance. The key 
difference between these two groups appears to be a recognition of the complexity 
involved in solving problems. Sorting the student work by multiple components helped 
these teachers to identify what was needed in a more precise way than by simply 
looking at correct or incorrect answers, but this practice did not necessarily reveal what 
students already knew or show how fo build on that developing understanding. Unlike 
the groups of teachers we discuss next, these teachers were not inclined to sequence 
different components of student performance in a progression that signaled different 
levels of achievement or development. 


Progression of Performance 


Much like the first two approaches, the six teachers who followed a Progression-of- 
Performance pathway saw formative assessment as a means of determining successful 
performance. Teachers in this category, however, attended to qualitative differences 
in student work and viewed performance on a spectrum; rather than simply identifying 
work as correct or incorrect, they categorized student work as more and less correct 
and incorrect, demonstrating an increasingly complex view of performance. Instead 
of viewing all correct responses (or incorrect ones) as the same, they differentiated by 
how close each response was to the most desired solution. For example, one teacher 
described an incorrect piece of student work as follows: “So they have a concept and 
understanding on how to arrive at the answer, and they set their problems up, but they 
were off computationally, by like two numbers, they just simply added or subtracted 
wrong... So the answer was close.” Thus, this teacher attended to the proximity to being 
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correct, rather than simply identifying the work as “incorrect” as in previous approaches. 

Like the previous group, teachers in this group attended to multiple components of 
performance in student work, but when sorting they ordered their piles of student work by 
perceived distance from the most desired performance. In describing her piles, a first- 
grade teacher explained: “they kind of get varying difficulties as they go. And there’s 
like, one, two, three... nine piles ...This is the furthest from the correct, closest to the 
correct.” 

Two teachers in this category assigned levels of performance to the work that were 
adapted from the four-level scoring guidelines used for state testing. Commonly referred 
to by teachers as a rubric, the score levels are used to designate “understanding of 
mathematical concepts and procedures required by the task” as either thorough (4), 
general (3), partial (2), or minimal (1) (Pennsylvania Department of Education, 2015). 
When asked how she determined that work was a 4 or thorough understanding, one 
second-grade teacher explained, “They showed all the work, and they both, they got 
the correct answer and they showed how they got the correct answer. So they had a 
very good understanding of how to solve it, how to set the problem up, and the answers 
were correct.” In contrast, students who demonstrated understanding of how to solve 
the problem but had a calculation error were given a 3 for general understanding, and 
students who left out a step in the process were given a 2 for partial understanding. 

Two additional teachers in this group sorted student work on a topic that had not 
been part of their summer training (e.g., fractions), and so found themselves applying 
the idea of looking for a progression of thinking and understanding to a content area 
for which they had no framework. As a result, these teachers had some focus on 
looking for gradations of understanding but did not know what evidence to look for 
and ended up creating progressions that reflected gradations of performance. For 
example, a third-grade teacher who was analyzing student work on a problem that 
asked students to compare visual models showing 1/4 and 2/9, sorted it in relation to 
the justification the student provided (“no reason or confusing reason” “all different 
reasons” and “connecting the idea that they know one fourth is equal to two eighths”). 
Acknowledging that she was unsure what to look for, she described asking herself, 

“What was the perfect answer? Maybe | should start there.” She felt that the use of 
equivalent fractions was an important component of the justification but was unsure how 
to evaluate other explanations that students offered and was not able to construct a 
progression of developing understanding. 

Most teachers in this category talked about instructional responses that were focused 
on getting students to the most desired performance, but the intensity of the response 
was in proportion to the proximity to the desired performance. As such, they did not treat 
all incorrect solutions as representing a complete lack of mastery. Instead, they thought 
about more intense responses for those who were far from the desired performance 
(e.g., needing to go back and learn foundational skills) and less intense responses for 
those who were very close to the desired solution (€.g., encouraging students to be more 
careful or check their work). 

This group of teachers appeared to have an overall view of learning as performance, 
but at the same time seemed to straddle achievement and learning orientations. While 
teachers attended to correctness and components of student answers, they no longer 
used a binary view when interpreting the student work. Teachers viewed the formative 
assessment process as a mechanism to assist in getting students to mastery, but by 
being more differentiated in their sorting process, they were able to more precisely 
tailor the intensity and focus of the instructional response. Thus, while there was some 
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focus on learning and understanding, getting to the most correct or desired solution or 
performance was still the ultimate goal of the formative assessment process. 


The Learning Orientation to Formative Assessment 


In contrast to the teachers and approaches described above, the remaining teachers 
in our sample sorted their student work by focusing primarily on the strategies students 
used to solve the problems, and then considered the correctness of answer and 

other components. Importantly, this focus placed emphasis on students’ developing 
understanding, regardless of whether the student had demonstrated the desired 
performance. However, we saw variation in the degree to which teachers understood 
and drew upon the developmental progression to analyze and respond to their student 
work. In describing these two variations, we begin with the approach that is most aligned 
with a learning orientation and reflects the approach that teachers were introduced to 
during the OGAP training. We then describe a variation we found on this approach that 
included some elements of an achievement orientation. 


Progression of Understanding 


This approach was characterized by placing primary emphasis on the strategies students 
used to solve the problem, using that strategy as evidence of the current level of 
understanding, situated within a developmental progression. The nine teachers in this 
group selected formative assessment items to learn more about what their students 
could do on different types of problems, rather than to assess what had been recently 
taught. As one teacher explained: 


[Students’] strategies will change depending on the question and how they 
understand the question. [Giving OGAP items] helps me understand, it helps me 
see maybe the type of question that they’re not necessarily understanding to the 
point where they go back to the additive strategies or they become in the non- 
multiplicative stage. It helos open my eyes to some of the specific things that they 
might be struggling with that I'm not able to see elsewhere. 


To sort and make sense of their student work, these teachers consulted the OGAP 
progression and sorted by the strategies that students used to solve the problem, 
creating multiple piles containing both correct and incorrect responses. In contrast to 
teachers previously described, they paid primary attention to how students were solving 
the problem as an indication of students’ developing understanding of number and 
operations. For example, they looked at whether students used ones, tens, multiples of 
ten, or base-10 understanding to count, add, or subtract to solve the problem. A few 
teachers in this category sorted student work on a topic not covered by the OGAP 
progression (é.g., interpreting data), but created their own developmental progression 
of strategies. The focus of their analysis was on how students got to the answer and what 
that suggested about students’ developing understanding. One first-grade teacher 
explained the benefits of sorting by strategy: 


| mean, it forces you to think about their way of thinking, rather than just right or 
wrong. What's their thought process? What strategy are they using? Who's higher 
on the thinking process, to be more quick and efficient than everybody else... like 
rather than the specific black and white, right or wrong answer. 
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In thinking about instructional implications, teachers drew upon their understanding 
of the development of that concept to build upon what students knew and move 

all students forward. A fourth-grade teacher talked about how he would respond to 
students who were solving a multiplication problem by drawing out equal groups and 
counting by ones: 


| would try to move them from... If they already have the circles, | would maybe 
have them put it in an array since they already feel comfortable drawing the 
circles. | would have them draw the same amount of circles if they want to in an 
organized array, and then once they understand the grouping of that array, I'd 
turn it into an area model and go from there. 


Like this teacher, teachers who used the OGAP progression often formulated instructional 
responses that focused on using specific visual models highlighted in professional 
development and on the progression, such as area models or open number lines, or 
focused on developing more advanced strategies. Some teachers described thinking 
about instructional implications as an inherent part of the sorting and analysis, rather 
than being a separate step. As one teacher commented, “| tried to group it... based on 
where they fell so | could see where they needed to go next.” 

Several teachers talked about using an 
instructional strategy learned in training (“select 
Teachers who sorted student work and sequence”) where they would select 
by strategy as a way to discern examples of student work from different levels on 
developmental levels of understanding In Pipa iee etn te leer aos ana meh 

: students make sense of and discuss similarities 
talked not only about how it helped between the strategies. One teacher explained, 
them determine what students “I'll put one or two pieces of work under the 
needed, but also how to help document camera and display it on the board 
and have the students tell me what they see, 
what they notice about it, and have them 
compare the different strategies.” 

The intent of this routine is to help students 
build on what they already know to understand 
and make connections to more sophisticated 
strategies. This focus on building on what students knew and could do to move their 
thinking and strategies forward incrementally was a distinguishing characteristic of this 
approach to formative assessment. Many teachers found this approach empowering, 
as it gave them a concrete way to think about supporting students. One fourth-grade 
teacher reflected: 


those students who seemed to be 
“getting it” continue to improve. 


Just seeing the progression and understanding the progression and knowing that 
if a student is not there, where do | need to go in order to get him to the next 
step? | don’t have to jump him all the way up to get him to where | want him. | 
could just focus on the next step and then slowly work him to there because the 
goal is at the end of the year where they should be, sooner, great, but a little at a 
time... At their progress, what they can, can’t do... But | think the one thing that | 
have learned is that it’s okay. If they didn’t get it, then that’s okay. | can figure out 
a way to get them to get it. 


Teachers who sorted student work by strategy as a way to discern developmental levels 
of understanding talked not only about how it helped them determine what students 
needed, but also how to help those students who seemed to be “getting it” continue to 
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improve. They saw improvement, rather than performance, as the goal of the formative 
assessment process, and differentiation as a way to benefit all students’ learning rather 
than only those who weren't meeting the expectations. 


Progression of Strategies 


Six of the teachers we interviewed used the OGAP progression and/or labels from 

the progression to describe student work; at the same time, their analysis of student 

work revealed a partial understanding of learning occurring in a progression. Like the 
teachers who sorted with a progression of understanding, these teachers gave formative 
assessment items to learn more about student thinking and/or students’ ability to do 
different types of problems. As one teacher described, “I was really interested in seeing 
that, diving deeper into how my students were thinking, so that | could help move them 
forward.” 

The focus on improvement and student thinking led teachers to put primary attention 
on the strategies students used, rather than the answers and components of the answers. 
For example, one fourth-grade teacher described sorting into two piles based on two 
levels of strategies from the multiplication progression (shown in Figure 1), multiplicative 
and early transitional, and then within those piles looking for “underlying issues,” which 
provided evidence of misunderstandings or errors. 

When sorting, these teachers connected evidence of students’ strategies to 
specific levels on the OGAP progression to make sense of the work. However, at 
times they demonstrated some misunderstandings in their use of the progression or 
difficulty integrating multiple forms of evidence in their interpretations. For example, 
one fourth-grade teacher characterized two incorrect solutions to the problem 118 x 
4 as multiplicative: one that showed the use of the distributive property with a minor 
error in the last step and one that showed the first step of a multiplicative strategy by 
multiplying 100 x 4. This second student was ultimately unable to use 100 x 4 to solve 118 
x 4 and ended up misusing the standard algorithm to get an answer of 1,132, suggesting 
significant misconceptions about multiplication and place value. The teacher focused 
on the strategy as the primary indicator of the level of thinking, but did not consider the 
nature of the errors in relation to whether the solution showed multiplicative thinking: 


If they used a particular strategy, and you can tell, you have evidence that 
they're trying to use that strategy, but it didn’t quite work out, or they started out 
using one particular strategy, and then they kind of got it mixed up alittle... at 
least they're thinking multiplicatively. At least they’re thinking that way, but they 
just had some underlying issues, which is one of the parts of the progression. 


In this case, by focusing on the strategy as the primary indicator of understanding, 
the teacher seemed to miss other aspects of student work, which revealed important 
misconceptions. 

As in this example, teachers who sorted using a progression of strategies were 
primarily focused on identifying and creating piles based on identified strategies but 
were not necessarily able to consider the larger holistic picture of the evidence of 
multiplicative thinking in the student work. They did not always understand the learning 
progression well enough to see how strategies reflected developing understanding 
and/or how the strategies built upon each other to build procedural fluency with 
understanding. Thus, while they may have ordered those strategies sequentially, guided 
by the progressions, it was not a truly developmental progression. Other teachers in this 
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category only partially or inconsistently used the progression to make distinctions among 
strategies. These teachers seemed to be making the transition to focus on strategy 

over answer, but in doing so were not able to focus on and integrate all aspects of the 
evidence in relation to the development of student thinking. 

The focus on strategies as an indicator of student thinking also led these teachers to 
use the location of the strategy on the progression to think about next instructional steps 
in terms of moving forward or addressing conceptual issues. Teachers who sorted using 
a progression of strategies viewed learning as a progression but sometimes had notable 
misunderstandings, which led to mischaracterizations of student understanding given the 
work presented. In some cases, they understood the need to move all students towards 
higher level strategies, but not necessarily how to build on what students already knew to 
get there. 

Despite drawing incompletely on the developmental progression of student thinking, 
the focus on evidence of student thinking was more important to these teachers’ 
analyses and instructional responses than the product or correctness of the answer. 
Teachers who sorted in this way also sometimes described how it changed their initial 
assumptions about student work. One third grade teacher remarked, “It shows me 
exactly where some of my kids are instead of where | thought they were.” 

Detailed case studies of teachers sorting and making sense of their own student 
work for instructional purposes, illustrating each of the five pathways, can be found in 
Appendix A. 


Discussion 


In this section, we discuss our findings in light of our research questions. First, we 
characterize the variations we found in how teachers made sense of their students’ 
work during the formative assessment process. We then consider what teachers’ 
interpretations and responses revealed about their underlying goals for student learning. 
Finally, we consider the implications of these findings for supporting teachers’ use of 
learning-progression-oriented formative assessment. 

Despite the fact that all teachers had been introduced to the OGAP progressions and 
a learning-trajectory approach to formative assessment, we found substantial variation 
in the ways they made sense of their students’ work and used it to inform their instruction. 
Most of the teachers in this sample looked beyond correct or incorrect answers when 
analyzing their own student work, and they looked at the work with an eye towards what 
they might do next to improve student learning or performance. However, as illustrated 
by Figure 3 and the descriptions above, we saw teachers taking different approaches at 
the different steps of the formative assessment process. Some teachers gave formative 
assessment items to see if their students could solve particular types of problems that 
reflected what had been taught, while others gave it to see what students could do on 
new and different types of problems. Teachers varied in what they paid attention to and 
prioritized in students’ responses as well as how they interpreted these features in terms 
of student learning. Sometimes their analysis resulted in a binary judgement, while other 
times it resulted in multiple categories that could be placed in a continuum. Finally, the 
focus of their intended instructional responses ranged from correction (showing them 
how to solve correctly) to differentiation (addressing specific needs) to development 
(moving to the next stage of level of understanding) to building on (using evidence of 
what students could do to continuously improve understanding). 
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It is important to note that all teachers in the study were drawing assessment 
problems from the same bank of items, provided by the program, and had access to 
the same content specific learning progressions. In many cases, the actual student work 
the teachers were analyzing reflected similar strategies and common errors. The stark 
differences and degree of variation found across the 32 teachers illustrates that simply 
providing teachers with similar training and resources will not necessarily lead them to 
take up formative assessment in the same way. 

Our analysis of the different pathways teachers followed to interpret student work 
illustrates the significant role that teachers’ goals for student learning play in the 
formative assessment process. Our findings surfaced two contrasting views of learning, 
reflecting the distinctions in learning goals offered by Dweck and Legget (1988). Teachers 
adopting an achievement orientation tended to focus on performance by prioritizing 
whether the solution met certain requirements, regardless of the strategy the child used 
or other hints the solution revealed about the student’s understanding. Teachers with 
a learning orientation instead focused on their students’ developing understanding by 
attending to the strategies students were using and the underlying concepts reflected in 
the use of those strategies. As Elliot and Dweck (1988) proposed, these indicators took on 
their own meaning and importance in the process of interpreting student work. 

The different pathways that surfaced from our analysis suggest that the formative 
assessment process involves making a series of decisions that establish, shift, or reinforce 
a learning or achievement orientation. At the same time, the three hybrid pathways that 
fall between the two extremes in Figure 3 reveal some dynamic aspects of teachers’ 
views or the possibility of some fluidity in these orientations at each decision point. For 
example, some teachers, at times, seemed to be moving toward a learning orientation, 
by looking beyond just the correct answer at multiple components in the student 
solution, but then were pulled back to an achievement orientation when interpreting 
those components in a way that involved a binary judgement. Limited understanding 
of the developmental nature of the learning progressions (i.e., how strategies build in 
sophistication as concepts develop and deepen) sometimes pulled teachers who were 
sorting by strategies back toward the more familiar achievement orientation, resulting in 
categorical distinctions that were differentiated but not developmental. 

Interestingly, the variations we describe here were not related to the self-reported 
frequency of use of OGAP items for formative assessment. Future analysis of the data 
collected for this study will focus on other internal and external factors that may be 
related to learning orientations and approaches to analyzing and responding to student 
work. 


Implications 


Our findings have several implications for teachers’ use of learning progressions for 
formative assessment. Our study shows that using formative assessment is not simply a 
matter of taking up new practices and using new tools. Teachers need opportunities to 
make explicit and critically examine their views about student learning in relation to new 
practices and goals for formative assessment. They may also need more time to make 
sense of the developmental trajectory underlying the OGAP progression, both in theory 
and in practice. The variations in take-up and understanding of the ideas that were 
offered in training, as reflected in teachers’ actual practices, suggests that it is important 
to provide opportunities for sustained learning and supported use over time. 
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The variety of pathways that resulted from decision-making during the formative 
assessment process highlights the complexity of analyzing student work for the purposes 
of informing one’s instruction, as there are several steps in the process at which teachers 
make interpretations: conceptualizing what they hope to learn from an assessment, 
what they focus on when looking at the student work, how they interpret the evidence 
in student work, and how they draw on that evidence to make instructional decisions 
or adjustments. Each of these decision points opens up or closes opportunities to view 
student learning as a process of achievement (performance) or continual growth 
(progression). At the same time, these decision points offer multiple footholds or points in 
the formative assessment process where teachers can begin to try out new approaches 
that reflect a shift in orientation to student learning. As Dweck (2015) is careful to point 
out, people tend to demonstrate a mixture of fixed and growth mindsets, and simply 
stating that one believes in a growth mindset is only the first step in a journey towards 
shifting beliefs and practices. Moving from an achievement orientation to a learning 
orientation involves a significant shift that goes against commonly held beliefs and 
practices of assessment. 

A final implication of our findings is to acknowledge that the achievement 
orientation is currently the default view of the most commonly used assessments in 
the U.S. educational system. In mathematics, both classroom-based and standardized 
assessments tend to focus on short, multiple-choice answers that can be used to 
calculate “percent correct” and make binary or normative judgments about student 
performance. Performance is communicated to students and parents as percentages 
and grades that encourage comparison to an externally derived standard and/or 
comparison with other students. The fact that these same test scores are used to make 
judgments about teacher performance reinforces the focus on ensuring that students 
demonstrate the desired performance rather than develop and grow over time. While 
most teachers may recognize the reasons why it is important to adopt a learning 
orientation, in practice they are pulled back into an achievement orientation. 

Our study also raises questions for further exploration. Are the approaches to sorting 
student work described here developmental? When teachers move from a singular focus 
on correct answers to consider multiple components of student work, does this set the 
groundwork for adopting a progression-oriented approach? Is creating a progression of 
performance a beginning stage of constructing a progression of understanding? These 
two pathways reflect significant differences in orientations—are we missing a stage that 
occurs in between that could help teachers make the shift towards prioritizing student 
strategies over performance? 

Finally, our data come from a single use of formative assessment, and it is important 
to note that we are not suggesting that this one instance reflects a complete picture of a 
teachers’ orientation to learning or learning goals. Although looking at this one moment 
in time helps us to highlight differences in approaches to formative assessment, we 
suspect that many teachers might show us a different approach with a different problem 
ona different day. Further study can illuminate whether and how teachers’ sorting 
practices are situated within their practice over time in a range of contexts, and also 
how those practices are influenced by both internal and contextual factors. 
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Appendix A: Case Studies 


In this appendix, we present case studies to illustrate the five 
pathways for analyzing students’ work that emerged from our 
analysis. We begin with two ends of the continuum, shown in the 
shaded rows in Table A.1 below: the correctness of answer and 
progression of understanding pathways. In many ways, these two 
pathways are antithetical to one another in almost every respect: 
the information teachers paid attention to in the student work, the 
implicit framework they used for classification of the work, and the 
implications they drew from the evidence for future instruction. We 
first explore these differences and then turn to examples of the three 
hybrid pathways to explore the push and pull between achievement 
and learning goal orientations to mathematics in the context of 
formative assessment. 


Table A.1 Characteristics of Case Studies Representing Five Pathways 


PATHWAY CASE ee SORTING INTERPRETATION RESPONSE 


Achievement 
Correctness of Answer Anne Can they do it? Answer Binary Judgment Correction 


Components of Bethany Can they do it? Multiple Binary Judgment Differentiation 
Performance Components 
Progression of Catherine Can they do it? Multiple Continuum Differentiation 
Performance Components 


Progression of Strategies Diana What can they do? Strategies Progression Development 


Progression of Elena What can they do? Strategies Developmental Building On 
Understanding Progression 
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Case Study A: Correctness of Answer 


We begin with the case study of Anne, who focused on the correctness of student 
answers as the primary source of evidence that her students were “getting it” in relation 
to the instructional goal. While her students used a variety of strategies to solve the given 
problem, Anne interpreted the use of those strategies as evidence that students were 
demonstrating what they had been taught to solve the problem successfully. 


Giving an OGAP Item “To see how they’re doing” 


Anne was a second-grade teacher with 30 years of teaching 
experience. She was in her seventh year of teaching at a small, 
relatively high-performing, K-5 school with a diverse student 
population (44% White, 21% Black/African American, 16% 
Hispanic/Latino, 6% Asian). For the interview, Anne brought wm (F] 
her students’ work on a result-unknown subtraction problem 
with two-digit numbers: “There were 91 apples at the store. 68 
apples were sold. How many apples are left? Show or explain 
how you know.” (91 — 68 = x) She explained that she had 
chosen the problem because her class had been working on 
two-digit subtraction strategies, and she “thought it would be a 
good one to see how they're doing.” She also liked using OGAP 
items because the textbook would direct students towards a STUDENT A 
specific strategy to use, but “this allows them to choose what 

strategy they want to use, which is better for me to see what 


a 
they're strong and weak on.” eh eee 
SE7O 

ae \ 


Figure A.1 Student Work 


Sorting Student Work by Correctness 


The student work that Anne brought reflected a wide variety 
of strategies for solving the problem, including algorithms 
and visual models. In sorting the work, however, she focused 
primarily on whether or not the work showed the correct 
answer, resulting in two large piles. The student work in the 
“correct” pile had all been marked with a large “C” on the 


paper while the work in the “incorrect” pile had been marked STODENE 

with a large “X.” As Anne looked through the piles and talked 

with the interviewer, she noted the different strategies they 

were using: “regrouping” (US standard algorithm), number = 
lines, base 10 visual models, “breaking apart” (decomposing 4 ~ 0 8 23 
by place value parts), and compensation. However, she did Y | 7 F 


not attend to the sophistication of thinking represented by = , } 
those different strategies in the OGAP Additive Progression. See PS 
Appendix B for how the progression distinguishes, for example, EZ 
between decomposing by tens and by multiples of ten, or the 
use of a visual model versus an efficient algorithm. 

The three examples of student work in Figure A.1 were all 
placed in the “correct” pile because they had the correct STUDENT C 
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Figure A.2. Student D’s Incorrect Solution numerical answer of 23. Anne did not seem 


concerned about whether the answer 
was labeled to reflect the context of the 
problem (23 apples). 
Each of these strategies shown in Figure 
7 A.1 can be located at different levels on 
aay i the OGAP Subtraction Progression. Student 


=~ 5 tens and then ones on a number line is an 
early transitional strategy, while Student B’s 


240" 
ih ee strategy of finding the distance between 
tg, 


> and Student C’s use of compensation to 
change the problem into 91 minus 70 and 
adjust the answer would be an additive 
strategy. However, Anne valued these 
strategies equally because the students 
obtained the correct numerical answer by 
using strategies that were taught in the curriculum. “They all used some strategy... The 
ones that got it right used different strategies, but | don’t think that they’re in a different 
hierarchy. Like, they are all strategies we've been dealing with, so, they just — Yeah, | 
didn’t get deep with it.” 

For the work in the incorrect pile, Anne identified whether the students were trying 
to use an appropriate strategy and then determined where the errors were. In the 
example shown in Figure A.2, Student D subtracted 76 on the number line instead of 
68. Anne noted that student set it up correctly (91 - 68) and may have been trying to 
use a strategy she had taught (the number line) but noting the errors in his strategy 
she concluded that he was not “paying attention.” She remarked, “and it’s a shame, 
because he’s got it set up properly”. In this case the use of the number line to subtract 
was an indicator that he was setting the problem up correctly. 


pie OME WIG EL 7) 9 


STUDENT D 


Instructional Implications: “Show them the correct way of getting it and 
the correct answer” 


When asked what she was planning to do as a result of giving this formative assessment, 
Anne focused on the work in the incorrect pile and the need to correct student errors. 


Maybe put them into small groups with all the same similar issues. Like all these 
ones that subtracted up and just show them like on a whiteboard or with base 
10 blocks, the correct way of getting it and the correct answer, and ‘Now look at 
your answer, what do you see here, can you tell why yours is wrong?’ You know, 
just to get them to stop doing that. 


Anne also explained that sorting in this way helped her determine whether students 
were “getting it” and whether she needed to go back and spend more time on a topic 
or strategy. As she said: 
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| think it’s good. | mean, it’s really diagnostic. It’s a good way of just ... You know, 
the topic test at the end of a chapter is a long time coming, so this is more open 
ended to the extent that you get to see where the students are strong and where 
they're weak or you know, you get to see what their thought processes are for 
how to solve problems. 


She also explained that giving formative assessment items helped to inform her pacing 
and preparation for the state test, which students would take in third grade. “Because 
you know, we don't have a lot of time, and some of them just aren’t getting the number 
thing and it’s like, okay, how do we fix it or how do we make them better,” she said. 
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Case Study E: Progression of Understanding 


Elena offers a contrasting case to that of Anne. Rather than focusing on correct 

or incorrect answers, Elena sorted her student work by strategy, using the OGAP 
multiplicative reasoning progression to interpret those strategies in terms of developing 
understanding, and then noting correct answers, labels, and other issues in student work. 


Giving an OGAP Item to See “What do students remember?” 


Elena is a teacher with over 20 years of experience. For the past four years she had been 
teaching fourth grade at a small K-4, low-performing school in a very economically 
distressed neighborhood with a primarily Hispanic/Latino population (62%). For the 
interview, Elena brought student work on a measurement conversion problem involving 
multiplication by a factor of 12: “Farmer Brown donated 7 dozen eggs to the senior 
center. How many eggs did he donate? Show your work” (7 x 12 = x). She explained 

that she and her grade partners had chosen to give this problem for two reasons: they 
wanted to see what students remembered about multiplication, and students were going 
to need to be able to use a given measurement conversion on the state test. 


Sorting Student Work by Strategy 


In contrast to Anne, Elena focused primarily on student strategies when sorting her 
student work. She referred to the OGAP Multiplication Progression (see Appendix C) to 
create three piles of student work in relation to the level of the strategy the student used, 
and then looked for errors or issues in the work. This resulted in work with correct and 
incorrect answers being in each pile. 

The first pile that Elena created was made up of student work that reflected the 
use of multiplication (7 x 12) to solve the problem. She noted that within this pile, some 
students had correctly labeled the answer as 84 eggs, but others had just written 84 or 84 
dozen eggs. She pulled out the example shown below in Figure A.3 and remarked it was 
“one of the best ones” because the student “multiplied correctly, got the right answer, 
then they wrote the answer... and they actually wrote everything out perfectly.” 
Although Elena valued student explanation, she considered the work that contained 
strong explanations as a subset of the multiplicative strategy pile. 


Figure A.3. A multiplicative strategy 
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Her second pile she described as “additive” because the students had “added 12 
seven times and got their number.” In both examples shown in Figure A.4, students added 
up 12 seven times. Student E had the incorrect label (84 dozens) and Student F had an 
incorrect answer of 74, which she described as “a simple error, but it’s still additive with 
errors.” 

The third pile she made was characterized by work that did not show evidence of 
understanding the multiplicative situation in the problem: 


It just represents students that didn’t seem to have any understanding of what the 
problem was asking them. They didn’t understand that they had to multiply and 
they didn’t understand that they had to subtract or add or anything. They didn’t 
know what operation to use, and just did all kinds of | don’t know what... So 

they just didn’t seem to have grasped the concept, or the understanding of the 
question. 


Figure A.4. Two additive solutions 
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As she talked about each of these examples, she looked for student sensemaking and 
understanding. In one example, the student had tried to represent 12 dozen eggs with 
representations of base 10 blocks but showed several misconceptions. Elena worked 
hard to make sense of what the student had done, noting “it was interesting what she 
was getting confused with. | think she was trying to picture something, but her picture 
was totally off.” 

When asked about what she learned from this assessment across the whole class, 
Elena focused on each of the three groups she had created from her sort. In reference to 
the non-multiplicative pile, she noted that there were “several that were really confused” 
and talked specifically about what they might need to develop understanding. For 
example: 


What she needs is to be able to see a dozen. | would love to be able to, for her 
it would be good to have even an egg carton that has 12 and then for her to 
actually see or even draw the 12 in each one, but | think seeing, visualizing the 
actual egg carton with the 12 and say you have seven of these. | think that in 
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itself would give her a better picture and better understanding and not get it 
confused. 


For the additive pile she remarked that “they struggled with the multiplication, so | see 
that they've resorted back to what they were comfortable with, and what they're usually 
comfortable with is addition and subtraction.” And for the students whose work showed 
multiplicative thinking, she talked about next steps for students who had not correctly 
labeled the answer (“I would probably work on making sure that they wrote 84 what?”) 


Instructional Implications: “Where I need to go in order to get him to the 
next step?” 


After analyzing the student work by sorting it by strategy, Elena explained how she had 
selected two examples of student work, one that was additive and one that showed 
multiplicative reasoning (Figure A.5), and then facilitated a whole class discussion about 
the work, both in terms of strategy and the meaning of the quantities: 

In reference to Student E's work, she noted that there were two issues she wanted to 
highlight, the incorrect label and moving towards a more efficient strategy, so she asked, 
“What's a quicker way? How many twelves did she put on there? What would’ve been a 
faster waye” 

With Student G's work she chose to illustrate a more efficient strategy as well as to 
highlight the same issue with the label. “So it reinforced it twice instead of just the one 
time.” She explained that she liked to show and discuss examples of student work to help 
build their confidence and sensemaking: 


A lot of them are like ‘Oh!’ Because they do know, and they know how to do 
it but it didn’t click, and so it was a good thing to see their response and their 
reaction when they realized that they did know it. [For] a lot of them it’s lack of 
confidence too, and that’s some of the ones that said ‘Oh.’ Because they did 
know it, but maybe because when they read it they didn’t understand it. 


Figure A.5. Two solutions selected by Elena for discussion 
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This focus on helping students recognize what they did know and understand was a 
theme across Elena’s responses in the interview. As she reflected on the benefits of 
sorting student work with the progression and then using the progression to select and 
sequence student work for discussion, she talked about how it had changed the way she 
was looking at students who “didn't get it”: 


Just seeing the progression and understanding the progression and knowing that 
if a student is not there, where | need to go in order to get him to the next step. | 
don't have to jump him all the way up to get him to where | want him. | could just 
focus on the next step and then slowly work him to there because the goal is at 
the end year where they should be, sooner, great, but a little at a time... At their 
progress, what they can, can’t do... But | think the one thing that | have learned 
is that it’s okay. If they didn’t get it, then that’s okay. | can figure out a way to get 
them to get it. 


As Elena describes, as a teacher, thinking about learning in this way and with the support 
of the learning progression gave her a greater sense of agency in terms of knowing how 
to help students move forward. She also reflected on the benefit of this approach for 
students in terms of supporting growth and improving their attitudes towards math: 


| think it’s very beneficial, especially to the students because they see, they do, 
know something. You give them credit for what they know because they got the 
right answer, it is 84, you just clarify it was eggs not dozens. So you praise them for 
what they do know just try to encourage them to improve or grow. 


Thus, Elena’s focus was on helping students recognize what they knew and understood 
as well as how they could continually improve. She saw learning math as a continual 
progression that included, but was not limited to, performance on a single item. 


Performance and Progression Orientations 


The approach taken by Anne and Elena differ in several respects. Both teachers gave a 
word problem; Anne chose one that was parallel fo what they had been doing in class 

and she gave it to see “what they’re strong and weak on,” while Elena gave a problem 
about something they hadn't focused on in a while “to see what students remembered 
about multiplication.” Both mentioned the fact that the knowledge they gained would 

be important for knowing how prepared students were for state testing. 

In sorting the student work, Anne placed primary importance on the correct 
numerical answer while Elena paid attention to the strategy students were using to solve 
the problem. In both classes, students used a range of strategies along the respective 
additive and multiplicative progressions, but for Anne, student use of strategies was 
part of the desired performance while for Elena they offered a window into the level of 
student understanding. For Anne, the sophistication of the strategy did not matter as long 
as it led to the correct answers. 

Both teachers paid attention to incorrect answers and tried to ascertain what caused 
the error. Elena's sorting by strategy resulted in correct and incorrect answers being in all 
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three piles. The treatment of errors had implications for the kind of instructional responses 
these teachers developed as a result of looking at the student work. Anne's response 
focused on students who were not meeting the standard for performance (which in her 
case was getting the correct numerical answer), and the emphasis was on reviewing or 
showing them correct strategies. Elena's response was to think about the needs of each 
group in terms of the appropriate next step on the progression, moving them forward 
from their current level of understanding towards more sophisticated strategies. 


Hybrid Pathways: The Push and Pull of Achievement Orientation 


The two pathways characterized above illustrate achievement (performance) and 
learning (progression) orientations that we observed in teachers’ analysis of student 
work, which in many ways are diametrically opposed. We also identified three hybrid 
approaches that incorporate aspects of both the progression and performance 
orientations. We see these hybrid pathways through the formative assessment process 
as illustrating ways that teachers might take up elements of a learning orientation, and 
in some cases begin to move towards using developmental progressions to analyze and 
respond to student work. In many cases, however, Movement toward a developmental 
approach appears to be constrained by a view of learning not fully compatible 

with this perspective, and/or a limited understanding of the developmental nature 

of learning progressions. In the case studies that follow, we present the three hybrid 
pathways — components of performance, progression of performance, and progression 
of strategies—beginning with the one most closely aligned with an achievement 
orientation. 
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Case Study B: Components of Performance 


Bethany is a case of a teacher who sorted her student work to determine who was 
“getting it” and who needed more support, but she also had an appreciation of the 
complexity of challenging problems and multiple facets in students’ responses. As a 
result, she did not look only for the presence of a single correct answer, but also for 
whether students showed their thinking and explained their answers. 


Giving an OGAP Item to “practice figuring out” 
and “show a model” 


Bethany was in her ninth year of teaching and sixth year of teaching second grade. 
She taught at the same school as Elena from Case 2, a small K-4 school in a primarily 
economically disadvantaged Hispanic population. For the interview, Bethany discussed 
her students’ work on the following problem: “Serena walked from her house to the 
grocery store. Then Serena walked 7 blocks from the grocery store to her grandmother's 
house. Serena walked 28 blocks altogether. How many blocks did Serena walk from 

her house to the grocery store? Show or explain how you know” ( x + 7 = 28). Bethany 
explained that the second-grade teachers had chosen this start unknown problem 
because they wanted to have students practice “figuring out what the problem was 
talking about” and “try to show a model.” She also described how she discussed the 
problem with students as a whole class first, in order to help them identify the important 
information and then model how to write it as a subtraction number sentence (28 - 7 = 
__). Rather than analyzing how they interpreted the problem (which could be solved with 
addition or subtraction), she focused on how they performed the subtraction that she 
had modeled. 


Sorting Student Work According to the presence of Multiple Components 


When she sorted the work, Bethany checked for a series of components in relation to 
her perception of their importance. First, she considered whether they used the correct 
operation (subtraction); then she considered whether they used visual models or 
drawings to support their solution; and finally she considered whether they had made 
errors in the use of the model. This process resulted in four piles as shown in Figure A.6. 

The first pile included student work where the students used addition, which she 
considered to be the wrong operation to use for this problem. Importantly, one of the 
two solutions in this first pile was correct, as the student had added 21 and 7 to get 28, 
but Bethany had not noticed this until the interview. The second pile included students 
who used only the U.S. standard subtraction algorithm to obtain the answer but did not 
show a visual model. The third pile included students who were trying to use a visual 
model - such as a representation of tens and ones blocks — to support their work, but had 
errors or misconceptions. The final and largest pile (n=20) included students who used 
correct visual models, whether it was to support their work or to figure out the answer. 

In some cases, the model used by students was correct but did not necessarily relate 
to their solution strategy. Figure A.7 shows an example of student work that Bethany 


Consortium for Policy Research in Education | WP 2019 — 2 


Pathways for Analyzing and Responding to Student Work for Formative Assessment 


Figure A.6. Bethany’s Sorting Strategy 
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initially put in the pile of “correct answers with a model.” The student drew 28 tallies under 
the tens column and seven in the ones, but then crossed out 18 from the tens and one 
from the ones, perhaps attempting to model the standard algorithm, which was written 
next to it. In describing the work, Bethany realized that the model was problematic and 
moved it to the pile with errors in the model. “Her tens and ones are off, so she would be 
someone else | would focus on a model with even though she has the right answer.” 

In her initial sorting of student work, Bethany seemed more focused on whether there 
was a model than how the student used the model to determine the answer. To her, 
the use of a visual model was an important component and an indication that students 
could explain their solutions, rather than an indication of what they understood about 
the problem. 

As she talked about this pile, she further divided the work by the type of model 
students used: number lines, tens and ones, or a picture where they modeled by ones. 
When questioned, she mentioned that number lines and tens and ones were more 
advanced models, but these distinctions did not appear to play a role in her sorting 
strategy. Further she also did not pay attention to how they were using those models, 

for example, whether they made jumps of one or 
Figure A.7. An exampe of a student ten on the number line, which would put them at 
who used an algorithm and a model different levels on the OGAP Additive Progression 
(See Appendix B). 


They all have the same answer, but they all kind 
of did it in a different way. So, whatever you 
are comfortable with as your model. So to show 
them that it can be done with tens and ones. 
It can be done with a picture. And then it can 
be done with a number line, but then we would 
_— even talk further about what way would be 
more... efficient for you. 
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Instructional Implications: “Let’s set it up with a model” 


Bethany's sorting illustrates her belief that visual models were an important means to 
understanding the problem situation, rather than evidence of developing understanding. 
Her instructional implications also focused on the use of visual models. She did not feel 
that any further instruction was needed for those students who had used an algorithm 
only to successfully solve the problem. “I think they understand the traditional algorithm, 
so | don’t know if | would even go back and say you have to do a model because they 
did understand what they had to do, so probably just continue regular instruction,” she 
explained. 

For the students who had made errors, she talked about having students “go back 
and figure out a model first”: 


Let’s look at a problem. Let’s set it up with a model, and they could even be 
able to pick, like, do you want to do the open number line? Okay, let’s see how 
it would look. If you want to do base ten blocks, let’s see how it would look. So 
just focusing on getting a model, so they can understand, at least, what they're 
supposed to be doing to help explain their thinking. 


For the students who had correctly used models, she described having them “talk about 
how they decided to solve it” and “think about what could be another way, another 
model they could use.” Her primary focus on the use of models as a component of 
students’ solutions appeared to help her determine whether students were “getting it” 
and whether she needed to go back and spend more time on a concept or topic. As she 
said: 

We have a pacing guide here in the district, so it’s like, ‘Gotta get it done,’ but 

then you look at the work, and it’s like, ‘Wow. | thought they were getting it, 

but this kind of shows that they didn’t, so how about we take a step back?’ So 

definitely [it] lets you see clearer. 


Bethany’s focus on the use of models along with the correctness of the answer may have 
helped her to put some focus on developing conceptual understanding, but this was not 
always focused on an accurate diagnosis of what students already understood. 
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Case Study C: Progression of Performance 


Similar to Bethany, Catherine also attended to multiple components or dimensions of 
student performance. But unlike Bethany, Catherine also adopted some aspects of a 
progression orientation by sequencing these components in a continuum. 


Giving an OGAP Item “To see what they could do” 
Catherine was a first-grade teacher with nine years of experience. She had been 


teaching first grade for six years at the same K-4 school where Elena and Bethany taught. 


For the interview, Catherine brought student work on a put-together/take-apart problem 
with three single-digit addends: “Nadia went to the store to buy vegetables. She spent 8 
dollars on lettuce, 3 dollars on tomatoes, and 2 dollars on carrots. How much money did 
she spend altogether? Show or explain how you know” (8 + 3+ 2 =x). 

Catherine explained that she had intentionally chosen this item because it was 
aligned with the work they had done on three addends but was slightly more advanced 
than the tasks they had done thus far. She said, “I wanted to see what they could do 
and where to go from there.” Her purpose for giving the item had some elements of a 
progression orientation, in that she was not only focusing on whether they were getting it 
but seeing how far they could extend their understanding. 

Catherine sorted the student work into nine piles, arranged in a continuum of 
performance, beginning with “furthest from correct” and moving towards “closest to 
correct.” Although correctness was a priority, she looked at multiple components of 
student performance to categorize the work, including whether they used the correct 
operation, added all three quantities, used a model, and wrote an equation. Further, 
she arranged these components into a continuum towards what she felt to be the ideal 
performance. Of the nine piles she created, four contained work with incorrect solutions 
and five with correct solutions. She described and ordered the piles according to the 
continuum show in Figure A.8. 

As shown in Figure A.8, Catherine sorted student work in relation to a continuum that 
moved from incorrect modeling of the problem, to using a visual model to represent 
the quantities in the problem, to solving the problem correctly with addition (without a 
model). She explained that the large number of piles was related to her thinking about 
instructional implications of their work: “Where | could go with that specific group of kids, 
what would be the next step.” 


Sorting by What to Do Next with Each Child 


In the example she selected from her first pile, at the lowest end of the continuum, 

the student had drawn 10 circles but after counting and labeling them correctly up to 
seven, wrote nine in the next circle and nine as the final answer. Catherine used this work 
to diagnose precisely what the student needed to work on: identifying the quantities 

in the problem, drawing a model to represent the quantities, numeral writing, and 
understanding the value of eight. 


| could see that he understood that he had to draw a model. To me, in my 
opinion, he understood that he had to put them together, because he did the 
circle. He didn’t separate them and put them in different piles... | put him by 
himself in this group because | knew that | wanted to work on writing. Here’s the 
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number eight, let’s draw a model for eight. And then being able to combine 
those three models within, into one. 


In another example from the fourth pile (correct visual model, but incorrect answer/ 
equation), the student had drawn 13 circles but then wrote the equation 1 + 2= 13 0n 
the page. Catherine categorized this solution as “incorrect” but noted both strengths 
and weaknesses in the work. 


| see that this student did again circle all three numbers. He had an equation of 
13 circles. Although they’re in one straight line, they're not in an organized, like 
eight, three, and two. There’s no way to tell where eight ended and three started 
and two started. He didn’t have any method to check off what he counted as 
he went, to keep himself on track. [He] has an equation with two addends and 

a plus sign, but has the answer of 13. But the numbers are one plus two, and one 
was not one of our numbers. 


She concluded that this student needed to work on “the organization of the model. 
Being able to show the eight, and then the three, and then the two. And then doing an 
equation that would have three addends with the correct numbers as the addends.” 

Finally the last pile, where students had added 8 + 3 + 2 = 13, she described as “just 
having the equation” and noted, “they knew it without the modeling. They knew the 
abstract.” 

The continuum created by Catherine’s sorting had some similarities to the OGAP 
Addition Progression (Appendix B), in that it moved from the use of a model to the use 
of more abstract reasoning. However, in her analysis, she was prioritizing performance, 
sorting by “what they were able to show me” rather than looking for the developing 
understanding that was represented by the strategy (e.g., counting by ones or using 
numerical reasoning). 


Instructional Implications: Differentiated Support 


Catherine's emergent continuum seemed to help her to see what each group needed 
to work on to improve their performance and proved useful for Catherine in developing 
instructional implications based on the evidence in the work. She explained that she 
worked with the whole class to compare solutions and emphasize what she was looking 
for in their performance: 


We looked at the ones with no equation versus an equation, or no model versus 
a model. And we talked about how just to be more efficient, or fo show your 
work, better than not showing your work. Because it helps us check it. It makes us 
do the right thing. It just makes it clearer when you're checking it to understand 
where the thinking was going. So we did talk about making models and making 
equations. 


She also described how she worked with a small group to address the underlying 
conceptual issues in modeling the problem situation: 


As a small group, I've talked to the students who were in pile three. The [students 
who used only] two addends, instead of three. We sat down and kind of worked 
a little bit deeper at getting three addends and having a model for all three, 
instead of just two. 


Notably, her instructional response is targeted at specific student needs, but lacks 
specificity in terms of how to develop that understanding. 
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Figure A.8. Catherine’s sorting strategy 
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Case Study D: Progression of Strategies 


Diana’s approach to analyzing and sorting student work incorporated many aspects 
of a progression pathway, including a focus on examining students’ strategies and 
their understanding. At the same time, she did not have a solid understanding of the 
progression from counting by ones to counting by tens, and she seemed unsure about 
how to use this information in her teaching. In some respects, she appeared to be 
learning through the process of articulating her thinking during the interview. 


Giving an OGAP Item: “I wanted to see if they could do it” 


Diana was one of two first-grade teachers at a small, low-performing K-8 school, where 
she had taught for more than 21 years in a 23-year career. The student population was 
100% economically disadvantaged and 90% Black/African American. In preparation for 
this interview, she had given her students the following start unknown item: “Some people 
were on the train. Then 12 more people got on. Now there are a total of 25 people on 
the train. How many people were on the train to begin with? Show or explain how you 
know” (x + 12 = 25). She explained that she gave her first-graders this item because she 
“wanted to see if they could do it with the missing addend. How many people were 
there to begin with? That’s always like a trickier thing.” Notably her analysis of students’ 
work seemed focused on determining whether they understood this aspect of the 
problem, as well as other information their work showed her about their understanding. 


Sorting Student Work by Strategy and Understanding 


Diana sorted the work into three categories as shown in Figure A.9. Each category 
indicated either something about students’ understanding of the problem situation or 
the strategy they used to solve it. Her description of student work in the first two piles 
illustrates a way that her analysis differentiated between errors of precision and deeper 
misconceptions. 


Figure A.g. Diana’s Sorting Strategy 
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One pile of student work that Diana created represented students who “didn’t get 
it.” These nine students made errors in how they set the problem up. Diana pulled out 
Student H’s work as an example (Figure A.10). “She drew circles and labeled them up 
to 12, which is the 12 more people got on the train. Then she drew 25 circles here, and 
the 25 was the total, but she did not realize that. She added these 2 together and got 
38. She missed the idea of a missing part.” It is worth noting that Diana did not comment 
on Student H’s minor miscalculation (12 + 25 = 38) but seemed more concerned that the 
student did not understand that the problem asked what number to add to 12 to get a 
sum of 25. 


Figure A.10. Student H’s work showing misunderstanding of the problem situation 


Eight additional pieces of student work were also in this pile. Several were harder to 
interpret than Student B’s. For some, Diana said the students’ strategy did not make sense 
to her. Others made detectable errors in interpreting the problem. 

In her second pile, Diana categorized the two students’ work as “gets concept but 
made a mistake.” Student J’s work in Figure A.11 shows two incorrect number sentences 
totaling 25. Instead of focusing on his errors, Diana noted that his equations showed an 
understanding of the start Unknown situation: “This one he ended up with the wrong, 
but... notin the way off, because this shows that he has an understanding of math. 

This is more like a precision problem.” The evidence in this work suggested to Diana that 
Student J understood that 12 plus the start value should add up to 25 and was trying out 
different start values. 

Looking at a checklist that was created from the OGAP Addition Progression Diana 
stated, “He’s closer to additive, and he just made a mistake.” (See Appendix B) 

Diana's third pile, which she characterized as “kids who all drew it... they all used 
circles” illustrates her attention to both strategy and the progression. Although the 
student work in this pile showed understanding of the problem and a correct solution, 
Diana expressed some concern that they all drew circles and counted by ones rather 
than using a more advanced strategy, such as a number line. When discussing Student K 
(Figure A.12), she referred to a recent test she had given. Student K had successfully used 
an open number line to show jumps of 10 and 1, to solve problems like 25 + 30. As a result, 
the student’s strategy of drawing 25 circles, crossing off 12, and counting the remaining 
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Figure A.11. Student J’s incorrect solution that shows understanding of the 
problem situations 
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circles by one puzzled and concerned her. 

This attention to strategy as an indicator of the level of sophistication of 
understanding is an important component of applying the developmental progression of 
additive thinking to formative assessment. 


Instructional Implications: “I want to move them up out of here” 


When asked to think about next steps for the whole class, Diana pointed to early 
counting level of the OGAP progressions for addition and subtraction and stated: “| 
want to move them up out of here.” Looking at the progression, she went on to explain, 
“Counting on from the first would be the next step .. .That’s still low but count on 
from the first. | think some kids can do that.” Although she wasn’t always referring to 
the progression in her sorting of student work, Diana was able to use it to identify the 
appropriate next level of strategy for most of her students. She went on to talk about 
the fact that in a recent test she had given her students, they were more successful with 
counting on, but she recognized that may have been a result of her telling them to count 
on from one of the numbers. At first, she found this to be conflicting information, but then 
concluded, “Maybe the reason that they didn’t do as well because they weren't told... 
Then telling them this was enough to make it work.” 

Diana was drawing sophisticated conclusions from multiple data points. On the one 
hand, she recognized that students were performing better on her own test because 
she told them to count on a number line. On the other hand, she recognized that this 
may have pushed them to try a more sophisticated strategy: “That’s like a legitimate 
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next step. Right? Telling them that number. Then the next step would be leaving that 
number off and seeing if they can do it on their own.” She also recognized that not all 
her students were ready for this push, as she talked about the students who hadn't shown 
understanding of the problem situation needing to start at a lower level: “| don’t know 
that these kids who came up with 54 and 17 and 39 are there... They need the circles, or 
they need actual counters.” 

Diana focused on whether students understood the ideas or could use particular 
strategies but had not yet mapped these strategies onto a discernable developmental 
progression. She demonstrated some familiarity with OGAP additive levels of 
understanding (i.e., recognizing that counting by ones was less sophisticated than 
making jumps of ten on a number line), but did not use them consistently to analyze 
her students’ work. She may have been developing her own tentative views of learning 
as a developmental progression, but she struggled to embrace this view to guide her 
teaching. At the end of the interview, Diana observed the following: 


| know that the OGAP was showing where a child’s number sense is. Right? The 
things they're saying for developing number sense take time. The problem is for 
them to go on to second grade with the number sense... it takes time to develop 
that number sense... | think it’s developing, but then | see this and | get scared. | 
think, wait a minute, it’s not developing like | thought... The only thing | can think 
of is maybe they'll go on to second [grade] and it will continue to develop. 


As this quote suggests, Diana appreciated the developmental nature of learning and she 
was beginning to think about how it could inform her instruction. She gave the formative 
assessment item to learn more about what students “could do on their own” and to 
uncover their developing understanding, rather than checking to see if they got it right. 
Although not completely adopting the OGAP approach, she was beginning to show a 
learning orientation towards formative assessment. 
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Progression—Addition 
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As students learn new concepts or interact with new problem situations and problems structures, they may move up and down along this progression. 
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Appendix C. Multiplicative Reasoning 
Progression—Multiplication 
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a Model 


ipa a a a a ik i a a a ay 


' Building up ' @ |e) eee 
Sa OOQS! OOS [eHsHs3]2 
1 3*3+343 el\eelele 
6 6 15 
6+6 ie 3 6 9 12 

1 1 Considers only ONE dimension Considers BOTH dimensions 
; : of an array or area model of an array or area model 


Na ea ea eee nv. a ee 


Additive Strategies 
Repeated addition with or without a model - 3 x 4=12 


@||e|e||@e 
waesea-12 Gogane) — [3/s/s|s 
3 3 3 3 @| ee) e 

Subitizing in small groups 3434343 


Early Additive Strategies 


Es 
= 
S 
3 
< 


Modeling, counting by ones Inconsistent Modeling, counting AK. 


Grouping by subgroups 
3x4=12 
Non-Multiplicative Strategies Underlying issues ind Eerors 


= 1 . ra ¥ 
Adds or subtracts factors Uses incorrect operation -Doesn't consider - Error in: calculation, place value, 


Models factors incorrectly Not enough information reasonableness of vocabulary, property or relation- 
solution ship, equation, or model 


The strategies students use move back and forth across the levels as they learn new concepts and/or interact with new problems structures and contexts. 


Unitizes into equal groups. uses visual models, understanding of place value, and properties of operations 


Guesses Uses procedures incorrectly 
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